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Abstract 


Currently, automation does not take into consideration the cognitive and emotional 
state of the crew. Rather, automation provides assistance based on explicit and static 
task assignments, with no adaptive capabilities, even though it is capable of providing 
higher or lower levels of support depending on the crew state and/or complexity of the 
operational situation. This chapter presents a new adaptive automation concept which 
offers an innovative ‘team’ centred approach to solving crew awareness/workload man- 
agement problems and enhancing flight safety. Partnership underpins the ‘Third Pilot’ 
approach. The crew (pilot flying and pilot monitoring), automation and the ‘Third Pilot’ 
are in charge together. Overall, partnership is proposed. This replaces existing paradigms 
involving dynamic changes in control function, where changes can be autonomously 
controlled by the system. Moreover, a new multimodal cockpit concept is advanced pro- 
viding enhanced assessment of crew state/workload. 


Keywords: adaptive automation, teamwork, workload, human factors, situation 
awareness, pilot decision making, stakeholder evaluation, cockpit 


1. Introduction 


Crew task support and information needs vary according to the crew composition, the crews own 
experience (i.e. familiarity with type, knowledge of the route), the specific flight situation (i.e. traf- 
fic and weather), and the crew state (i.e. fatigue, confusion) [1, 2]. With increasing duty time and 
traffic growth, pilots can benefit from an “experience aid’. Ideally, the crew and the ‘experience 
aid’ (or assistance system) comprise a cooperative system [1, 2]. This cooperative system approach 
follows the cognitive systems engineering frameworks, as advanced by Hollnagel and Woods [3]. 
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In its current form, automation does not take into consideration the pilot’s cognitive and emo- 
tional state. Rather, automation provides assistance based on explicit and static task assign- 
ments, with no adaptive capabilities, even though it is capable of providing higher or lower 
levels of support depending on the crew state and/or complexity of the operational situation 
[1, 2, 4]. It is argued that safety is optimised when human and automated systems adapt both 
to each other and to the specific operational context. This guarantees fluent and cooperative 
task achievement— maintaining safety at all times. 


This research was undertaken as part of the Applying Pilots’ Model for Safer Aircraft 
(A-PiMod) project, funded by the European Commission (Framework Programmes for 
Research and Technological Development) [1]. The aim of this project was to demonstrate a 
new approach/concept (and associated technologies) for an adaptive automation and multi- 
modal cockpit which might mitigate and/or reduce human error. The project commenced in 
September 2013 and finished in September 2016. 


2. Background 


2.1. Automation 


Currently, Pilots share responsibility for different flight tasks with cockpit systems. As defined 
by Kaber and Prinzel, adaptable systems are systems which require human delegation of task 
and ‘function authority’ to automation during real-time operational performance (i.e. the task 
distribution is controlled by the user) [5]. This is different to adaptive automation (AA), which 
allows for dynamic changes in control function allocations between a machine and human 
operator based on states of the collective human-machine system’ [6, 7]. 


Human factors problems with automation have been cited as contributory factors in many 
air accidents. This includes: Flight Air France 447 (2009) [8], Flight Spanair 5022 (2008) [9], 
Flight Helios Airways HCY 522 (2005) [10], Flight China Airlines 140 (1994) [11], and Flight 
Air Inter 148 (1992) [12]. The air accident reports highlight several human factors issues such 
as automation surprises, reduced situation awareness, workload problems and over-reliance 
on automation [1]. 


2.2. Crew errors 


Errors are defined as ‘actions or inactions by the flight crew that lead to deviations from 
organisational or flight crew intentions or expectations’ [13]. Unmanaged and/or misman- 
aged errors frequently lead to undesired aircraft states. Flight crew errors reduce the mar- 
gins of safety and increase the probability of adverse events [13]. As documented by the 
International Civil Aviation Organisation [14], human error is a causal factor in between 
60%-80% of accidents and serious incidents. It is stated in the ‘Flightpath 2050, Europe’s 
Vision for aviation’, that ‘the occurrence and impact of human error’ will be ‘significantly 
reduced through new designs and training processes and through technologies that support 
decision making’ [15]. 
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2.3. Theoretical starting point 


In line with a cooperatives systems approach, the question of automation status/communica- 
tion and its role in the execution of the flight, should to be framed from a team perspective 
and linked to a risk assessment of the mission and the selection of a course of action. Thus, the 
theoretical starting point for addressing human factors issues with automation (specifically, 
teamwork, task distribution, authority, situation awareness, error detection and workload 
management), is the assessment of the crew/automation/aircraft/environment state in rela- 
tion to the achievement of the mission level goal (i.e. mission level risk assessment), and the 
identification of a suitable task distribution at cockpit/agent level, to achieve this [1, 2]. So con- 
ceived, automation has a role in relation to (1) real-time risk assessment, (2) the identification 
of a course of action, (3) the selection and subsequent implementation of a course of action, 
and (4) the identification of an appropriate task distribution based on the crew state [1, 2]. 


Further, it is argued that there is relationship between addressing human factors issues with auto- 
mation (specifically, teamwork, situation awareness, task distribution, authority, error detection 
and workload management) and improving crew interaction with cockpit systems. The provi- 
sion of amultimodal concept can support the above. In addition to (1) allowing for flexible inter- 
action with cockpit systems and (2) providing a means to communicate with the crew in relation 
to crew state and decision support, the multimodal cockpit has a role in relation to (3) supporting 
the better assessment of crew state/workload (information inputs re crew activity/interactions). 


2.4. Methodological background 


Stakeholder involvement in programme evaluation has been recognised as one of the most 
effective approaches to enhancing the use of evaluation findings, and ensuring the validity 
of the evaluation activities [16]. Stakeholder involvement is defined as the participation of 
(programme) stakeholders in any phase of an evaluation [17]. Stakeholder involvement can 
vary with regard to diversity in stakeholder selection for participation, the control of techni- 
cal evaluation decisions and the depth of stakeholder participation in the programme/project 
evaluation process [18]. Stakeholders are conceived as invaluable source of knowledge, per- 
spectives, information on context and needs. Drawbacks of stakeholder involvement are also 
reported. This includes the feasibility of implementing a successful participative study. For 
example, time, cost, involvement from (disadvantaged) groups and skills required from an 
evaluator in facilitation and ‘good listening’ [19]. 


The involvement of stakeholders to accomplish given tasks by participating in common activi- 
ties has been central to ‘Community of Practice’ concepts [20]. ‘Community of Practice’ members 
engage in a set of relationships over time around some particular area of technical knowledge 
or skill associated with the given tasks. This allows the members of a specific ‘Community of 
Practice’ to generate a sense of joint enterprise and identity by sharing a practice— doing things 
together, developing a sense of place, common goals. In Wenger's analysis, three characteristics 
are crucial to define a ‘Community of Practice’: (1) the ‘domain’ — which specifies the identity of 
COPs with the specific competence and commitment the stakeholders engage; (2) the ‘commu- 
nity’ — stakeholders build their relationship interacting in joint activities, sharing information 
and common objectives and learning from each other; and (3) the ‘practice’ —stakeholders share 
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a repertoire of resources (experiences, stories, tools, ways of addressing recurring problems) 
which help forming the practice with time and sustained interaction [20]. 


3. Research design 


3.1. Objective 


The aim of the A-PiMod project was to demonstrate a new adaptive automation concept 
(and associated technologies) enabled by a hybrid of three elements: (1) multimodal pilot 
interaction, (2) operator modelling and (3) real-time risk assessment. It is anticipated that the 
introduction of this new automation concept will reduce human error— making substantial 
progress in relation to aim of reducing the accident rate by 80%. Table 1 below provides a 
description of the high level project goals 


3.2. Overview 


The overall design/evaluation methodology combines formal HMI design/evaluation activi- 
ties (i.e. interviews and simulator evaluation), informal HMI design/evaluation approaches 
(i.e. participatory design activities), along with an integrated stakeholder approach to evalu- 
ation [4]. Overall, 23 COP sessions and two phases of simulator evaluation have been under- 
taken. The first phase of simulator evaluation involved eight participants, while the second 
phase involved 12 participants. This has been reported in more detail in [1, 2]. 


3.3. Community of practice 


The concept of ‘Community of Practice’ as proposed by Wenger [20] underpins the A-PiMod 
‘Community of Practice’ approach. The A-PiMod ‘Community of Practice’ involved stake- 
holders who shared technical knowledge and skills associated with relevant functions in the 
Air Traffic Management (ATM system), and broader aviation related domain. Overall the 
role of participants in the A-PiMod ‘Community of Practice’ concept was characterised as a 
‘participatory’ approach. Members engaged in a range of validation/evaluation activities on a 
continuous/regular basis, through the run-time of the project. 


The panel of stakeholders in A-PiMod included both “primary users’ (i.e. internal stakeholders 
representative of each project partner), and ‘all legitimate groups’ (i.e. external stakeholder 


No A-PiMod high level project goals 

1 To reduce accident rate by 80% 

2 To achieve a substantial improvement in the elimination of and recovery from human error 
3 To mitigate the consequences of survivable accidents 


4 To support smart pilot assistance 


Table 1. A-PiMod high level project goals. 
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representative of the aviation-related industry and Flight operational system). Stakeholder par- 
ticipation involved consultative interaction along with engagement in technical research tasks. 


Figure 1 below depicts the composition of the community of practice — with attention to the 
levels of expertise of both internal and external stakeholders. Internal stakeholders are repre- 
sented in blue. External stakeholders are represented in amaranth. The overlapping levels of 
expertise are indicated by the red dotted line. 


3.4. Simulator evaluation 


Two sets of simulator evaluation were undertaken. In both cases, the evaluation involved a 
mixed-method approach with the administration of semi structured interviews, simulator obser- 
vations, collaborative workshop sessions and questionnaires. Each set of evaluations involved 
two crew members and elapsed over 2 days. Overall, a scenario-based evaluation approach 
was adopted. Simulator evaluation involved the use of the DLR simulator—the GECO system. 


In day 1, participants were first briefed about the overall procedures, consent was obtained, 
and profile information was elicited. Then participants undertook a semi-structured inter- 
view regarding the current state of automation and HF still-open issues. After this both 
participants obtained training, in relation to interacting with the new adaptive automation 
multimodal concept/technologies in the simulator. The training was delivered with the sup- 
port of slides, software training tools, and some hands-on training in the simulator (i.e. using 
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Figure 1. A-PiMod Community of Practice: Stakeholder Competency and Knowledge (Source: Cahill et al. [2]). 
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the A-PiMod user interface displays such as the MC-M Display and the MultiModal ND). 
Then, hands-on training in relation to using the GECO simulator was provided. After this 
training session, participants undertook the specific simulator evaluation session, addressing 
the adaptive automation and multimodal concept/technologies. After the simulator sessions, 
participants were asked to complete a questionnaire evaluating the adaptive automation and 
multimodal concept and technologies. Further, a specific simulator session was undertaken 
with a focus on the interaction with the MultiModal ND. This involved a collaborative session 
to evaluate the operational and safety aspects pertaining to the multimodal cockpit. 


In day 2, participants were asked to evaluate the adaptive automation and multimodal con- 
cept and technologies. This involved individual post evaluation individual semi-structured 
interviews. Then a collaborative workshop session was undertaken. This focussed on three 
distinctive scenarios (i.e. (1) supporting routine operations, (2) avoidance of conflict of taxi/ 
way or apron, (3) Incapacitation (VETO)) and to what extent A-PiMod could support the 
Pilots in relation to workload management, error identification, situation awareness, team- 
work, and how this would have an impact on the error reduction. After this, the participants 
completed an individual questionnaire related to benefits analysis. 


3.5. Assessment of benefits/safety impact 


In order to assess the potential safety impact of the new automation concept (and allied multi- 
modal cockpit), a systematic approach was applied. This involved the elicitation of structured 
feedback from pilots and experts, using the Total Aviation Risk model [21]. Specifically, struc- 
tured feedback was captured in relation to potential change factors for base events in this risk 
model [21]. The total aviation system risk model contains 425 base events and 51 end states. 
The particular structure of an event sequence diagram and its connected fault trees depends 
on the scenario considered. The probability change factors of all impressionable base events 
are determined in the safety assessment on the basis of information gathered in multiple 
workshops with members of the Community of Practice (COP). In the workshops the par- 
ticipants discussed the kinds of mechanisms facilitated by the innovative concept which may 
increase or decrease the probability of a particular base event in a scenario. The specific details 
of the quantification of safety impact have been reported in another paper [22]. 


4. Proposed adaptive automation concept 


Overall, the objective is to provide assistance to the flight crew when and if required. 
Automation acts as a third crew member providing information and task support to crew 
-safeguarding the mission level goal [1, 2]. 


The third pilot approach involves providing (1) decision/information support, and (2) work- 
load support. This follows from the hypothesis that information underpins good decisions, 
which in turn results in safe aircraft states. Further, it follows the philosophy that if there in 
increase in workload, certain functions should be shifted to automation, to reduce the cogni- 
tive/physical burden on the crew. 
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The overall approach involves continuously monitoring the operational situation and the 
allied crew/automation state, to determine the best distribution of task activity between 
the crew and automation [1, 4]. It is anticipated that this will ensure the safe completion 
of the flight. If there is an increase in workload, certain functions will be shifted to auto- 
mation, to reduce the cognitive/physical burden on the crew. Also, automation is used 
to support information management tasks (i.e. information gathering and information 
assessment). This in turn has an impact on safety. Real-time feedback is provided to the 
Flight Crew via a cockpit user interface (i.e. HMMI Interaction Manager), so that the crew 
at all times understand the status of (1) the operational situation, and (2) the joint crew 
automation system. This ensures full situation awareness, which in turn impacts on mis- 
sion safety. All of the above ensures that the aircraft remains in a safe state. This in turn 
has consequences at a process level (i.e. both single and multiple flight levels). 


The team comprises the Flight Crew (namely the Pilot Flying and the Pilot Monitoring), the 
‘Third Pilot’ and automation. Accordingly, the third pilot is conceived as a virtual team- 
member [1, 2, 4]. All team members co-operate in relation to making and executing mission 
level decisions. Mission level decisions are enacted at the mission level and translated into 
new cockpit level tasks that have to be distributed between the crew and automation, and 
then performed by them. The system continuously monitors the operational situation and the 
allied crew/automation/aircraft state, to determine the tasks the team has to perform together, 
and how to best distribute them between the crew and automation. 


The new cockpit technology (i.e. automation and associated systems) allows us to answer the 
following questions: 


e Is the joint crew/automation system in a safe state? 
e Is there a potential for a safety critical aircraft state (i.e. now and/or the near future)? 


e Do the crew (Pilot Flying and Pilot Monitoring) require support in terms of increased levels 
of automation? 


e Do we need to adjust the level of automation? 


e Do the crew require information/decision support? 


A-PiMod flags potential risks— providing operational guidance in relation to managing those 
risks. Pilots have final control, but are responsible and accountable for their decisions and 
actions [1]. The crew are responsible for assessing the risk status of situation and the appro- 
priate course of action. As such, the crew are not required to follow the decision support 
provided by the third pilot. This decision support is an aid, not a requirement. The crew can 
over-ride system proposals/decisions, except in certain critical situations (i.e. incapacitation). 


A-PiMod adopts a team centred approach as opposed to a crew centred approach. Specifically, 
is assists the Flight Crew in relation to information and workload management (i.e. interven- 
tion if over and/or under loaded). Automation is conceived as an extension of the pilot’s ability 
to carry out an action(s). Automation provides decision and information support. In principle, 
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we are focussing on the outcome. That is, we are considering what is best for the safe and 
efficient completion of the mission, as opposed to adapting to human needs. As indicated in 
the architecture concept (see below), if the Pilot Flying/Pilot Monitoring is overloaded and this 
threatens the completion of the mission, the task distribution is adapted at the agent level [1, 2]. 


As reported previously, we propose partnership as opposed to dynamic changes in control func- 
tion (such that changes can be managed autonomously by the system) [1]. The A-PiMod architec- 
ture describes the means for the adaptive completion of Mission Level tasks and their distribution 
between the crew and automation. This includes the real-time analyses of both the pilot’s state 
(situation awareness and workload) and mission risks [1]. Apimod will permanently assess 
what has to be done by the cockpit (mission and its context = > cockpit level tasks), distributing it 
between the crew and automation, and assessing if these agents are correctly performing the tasks 
they have been assigned (i.e. recover from a stall, avoid ground obstacles, etc.). Task distribution 
is the product of a situation management process [1]. Here, the pilot flying and pilot monitoring, 
along with other automated processes cooperate to assess the situation, its risks, what has to be 
done (cockpit level tasks) and the associated risks [1]. This results in a task distribution. 


As described in the architecture concept, in situations where the flight crew are overbur- 
dened, task distribution is adapted at the agent level [1]. Automation adapts to crew states 
and capabilities, so that all required cockpit-level tasks are performed [1]. It is this architec- 
ture that guarantees the safe completion of the flight. 


The proposed automation concept addresses the key decision requirements as defined in the 
safety case [1, 2, 4]. For more information, please see Table 2 below. 


No Decision How supported by automation 
requirement 
1 _ Authority Boundaries for automation set in relation to role of Pilot and associated decision authority 
2 Information Supports information acquisition and analysis 
3 Time Automation provide real time updates as to status of situation (both current and future) so 


have time to anticipate potential future problems and how might manage them 


4 Judgement Automation provides decision assistance — providing feedback on potential course of 
actions and risk associated with each. 


Automation can gather decision support information from actors outside the cockpit — if 
collaborative/consultative decision 


5 Teamwork Human agents/crew and automation are conceived as a team 


Better collaboration between team members - enhance situation awareness and increase 


safety 
Cockpit as a cooperative system 


Support teamwork in terms of information sharing: (1) information sharing between crew 
and automation, and (2) information sharing between cockpit and agents outside the 
cockpit 


6 Feedback Automation provides feedback (updated information picture) in relation to the outcome of 
decisions taken and next steps 


Table 2. Automation provides support for decisions. 
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Underpinning the Third Pilot concept is the A-PiMod multimodal cockpit concept [1, 2]. The 
multimodal cockpit (1) allows for monitoring of crew interactions with cockpit systems, (2) 
supports crew communication with automation (i.e. via the MCM Display) and (3), enables 
voice and touch interaction with cockpit displays [1, 2]. 


5. Proposed technical architecture 


The A-PiMod architecture allows for (1) adapting the mission based on current circumstances 
and (2) the subsequent organisation of the cockpit (task distribution between the crew and 
automation) and (3) the circulation of information between the crew and cockpit systems 
(including automation) to the current and future situation(s) [23]. The starting point in the 
architecture is to adapt the mission permanently, and to translate a given mission configura- 
tion (e.g. decision to divert to closest airport) into tasks that have to be executed by the ‘team’. 
The tasks are then distributed between the team members (crew and automation) based on 
their current states and capabilities. 


Figure 2 below depicts the A-PiMod architecture. This is based on a three layer hierarchy of 
tasks [23]. 


Mission Level 


Tasks 


Mission context 


Cockpit Level 
Tasks 


Cockpit context 


Agent Level 


Tasks 


Figure 2. Task Hierarchy. 


12 Aircraft Technology 


The architecture is a ‘control’ architecture; it’s about the distributed control of a given object. 
In this example, this refers to the overall aircraft state including navigation and trajectory. 
Tasks at a given level are translated into the tasks of the level below based on the context 
of their execution [23]. At the mission level, the context of execution includes (1) the state of 
the aircraft and (2) the traffic and environmental context in which the aircraft is flying (i.e. 
weather, ATC, traffic). At the cockpit level, context refers to (1) the status of the cockpit agents 
(namely pilots and automation) and (2) cockpit equipment (i.e. displays). 


As detailed in Table 3 below, this control architecture is elaborated in relation to three levels. 
This includes the mission level, the cockpit level and the agent level. These levels are hierar- 
chical—each level is a decomposition of the level above. All actions occur at the agent level. 
The upper levels are about deciding what tasks the different agents perform, based on the 
current contexts, at the mission and cockpit levels. 


The key level is the cockpit level. We are trying to see what the cockpit as a whole (i.e., all of 
its agents—be they human or machine) has to do at a given point to achieve the Mission Level 
Tasks (in the current situation). Then, what the cockpit as a whole has to do is distributed 
between the agents available in the cockpit. 


It should be noted that the broader ATM system level is not explicitly referred to in the archi- 
tecture. This is because the other aircraft are not under the control of the architecture. They are 
part of the context in which the architecture is flying (i.e. achieving its control tasks). 


As indicated in Figure 3 below, several technical components have been specified, linking to 
the overall architecture concept. 


As outlined in Table 4 below, the A-PiMod architecture consists of seven components and 
two separate software systems [22]. 


A key feature of the architecture is the distinction between components and modules: 
e A component is a functional unit that performs specific tasks (e.g. risk assessment at the 


mission level). In the A-PiMod architecture a component is always a small cooperative 
system made of the crew +1 module. 


e A module is a software system that acts with the crew as a team in this specific functional 
unit. The module provides assistance to the crew in the performing the component’s tasks 


Level Description 


Definition of mission level Single flight level— mission phase/sub-phase, context, A/C state—the context the 


tasks aircraft is in 

Definition of cockpit level Tasks that need to be performed by all (i.e. crew and automation) to achieve mission 
tasks goals given specific context elaborated 

Definition of agent level Actions undertaken by crew or automation to achieve mission goals in specific context 
tasks 


Table 3. Architecture levels. 
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Figure 3. A-PiMod Architecture (Source: Cahill et al. [2]). 
Component 1. Situation determination @Mission Level 


2. Risk assessment @Mission Level 

3. Situation modification@ Mission Level 

4. Task determination @Cockpit Level 

5. Situation determination @Cockpit Level 

6. Task distribution @Cockpit level 

7. Risk assessment @ Cockpit Level 
Separate Software Systems 1. Crew state inference 


2. HMMI interaction manager 
Table 4. A-Pimod architecture. 


Thus regarding risk assessment at the mission level, it will always be performed together by 
the crew and the dedicated module, acting as a small team for that purpose. Both the crew and 
the module have the common goal to achieve the function assigned to the component. This 
is what brings great flexibility to the architecture. It allows each component to be executed 
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exclusively by the crew (‘manual’ mode), e.g. the crew solely assess the risks, by the module 
(‘automated’ mode), e.g. the module solely assess the risks and the crew has no say in that 
evaluation, or by both the crew and module (‘mixed’ mode), e.g. the module produces a risk 
assessment and the crew acknowledge or reject it. This is true for all seven components in 
the architecture. Each is made of the crew and a dedicated module. Given the crew can par- 
ticipate to all seven components, the crew superposes the seven functions associated with 
the individual components, and this is exactly what human pilots do. They perform all these 
operations permanently, without being aware of that dissociation between seven elemen- 
tary functions. They are thus part of the situation assessed by the Situation Determination/ 
Modification @Mission Level component. All these tasks are permanently revised, distributed 
and executed, based on the context. 


In addition, visual analysis of pilots’ behaviour is recorded to infer human operator’s (pilot's) 
mental state, stress level, and general workload. The following instruments/technologies are 
used to obtain information about the pilot’s behaviour: 


e Eye tracking 
e Gesture recognition 


e Head pose 


6. Pilot interaction in the cockpit and associated user interfaces(s) 


6.1. Overview 


Crew interaction with cockpit systems is simple and user friendly. In addition to traditional 
controls, pilots interact with the system using voice and touch. This interaction is tracked by 
the system (i.e. what tasks performing, level of fatigue, involvement in activity). This form 
of tracking is referred to as ‘crew state monitoring’ [1]. Crew feedback is provided via a new 
cockpit user interface (Mission and Cockpit Level Management Display —MCMD). This 
provides information about (1) the risk status of the operational situation (this includes an 
assessment of the status of joint crew/automation system) and (2) what to do (including the 
provision of best options/alternatives based on different ‘technical’ contributing factors such 
as fuel remaining, the weather status at alternate airports and so forth) [1, 2]. As noted previ- 
ously, the crew can over-ride system proposals/decisions, except in certain critical situations 
such as crew incapacitation. 


6.2. The A-PiMod MC-M display 


The A-PiMod Mission and Cockpit Management Display (MC-M Display) enables commu- 
nication between the Flight Crew and the new adaptive automation technologies [1]. The 
MC-M Display supports both mission and cockpit management tasks. As depicted in Figure 4 
below, the proposed MCMD features two related sub-displays—the mission and cockpit level 
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Figure 4. The MC-M Display (Source: Cahill et al. [1]). 


displays [21]. In the cockpit level display, one can see the tasks assigned to the crew and to 
automation. The software allows the crew to change that distribution—both manually (crew) 
or automatically (automation). In keeping with concepts of authority/pilot control, all task 
distribution changes have to be approved by the crew before becoming active. 


6.3. Multimodal interaction 


During the A-PiMod project a multimodal system prototype was developed to explore possibili- 
ties of multimodal interaction in context of flight deck. From the perspective of human-machine 
interaction, the flight deck interfaces should be able to accommodate large number of diverse tasks 
while maintaining high level of efficiency, usability and uncompromised safety. In A-PiMod mul- 
timodal interaction was demonstrated through the Multi-Modal Navigation Display (ND) [1, 2]. 


A number of other systems are linked to the crew state estimation and crew task determi- 
nation components, for the purpose of inferring (1) the pilot’s mental state, stress level and 
general workload, and (2) predicting potential errors, missed events and/or overlooked infor- 
mation. This includes eye tracking, gesture recognition and head pose tracking technology [1]. 
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7. Crew state monitoring 


Crew state monitoring (that is, focussing the Pilots attention on their state along with that 
of their crew member—and on the current and future state of the aircraft) is an important 
function of the third pilot. If fatigued, pilots may either forget or omit to review all the safe 
options. The 3rd crew member (automation) will consider all safe options and prompt the 
crew in regard to possible options, to ensure that a safe decision is made. In this context, a 
key challenge is how to get the two human crew members to share their ‘current state’ with 
the 3rd crew member [1]. Such an exchange should be meaningful and informative but not 
self-incriminating in any post hoc analysis [1]. Normal human interactions can easily accom- 
modate this. For example, such information might be imparted as part of pre-flight social 
interactions/conversation. However, this is hard to replicate in relation to human/machine 
(i.e. third pilot) interaction 


The assessment of crew state goes beyond issues of workload. It concerns many factors 
including crew experience, flight hours, crew familiarity with the proposed route and depar- 
ture/landing airports, training background and so forth. The crew state might be assessed as 
less optimal in situations when the two crew members are unfamiliar with the route. From an 
operational/Pilots perspective, the starting point for crew state monitoring is the crew brief- 
ing/flight planning. Depending on the crew situation, this might occur a week before the 
flight and/or at the time of the pre-flight, flight planning and briefing task. In addition, knowl- 
edge of the joint crew status is and any issues linked to this is required [1, 2]. Potentially, the 
crew might need to report their status/state before the flight commences [1, 2] 


The means/basis by which crew state information (i.e. eye tracking data, gesture data, voice, and 
touch data) is used to assess the crew state needs to be carefully considered. The assessment 
of this data depends on what we already know about the crew (for example, location in roster 
and expected level of fatigue). For example, if the crew are not looking at the correct area of the 
screen, and/or are blinking their eyes a lot, it may not be a problem. In this case, the crew might 
be very familiar with the route and may be more or less fatigued (i.e. location in roster—first 
day or last day). However the system might interpret this behaviour differently if the crew are 
unfamiliar with the route, and on the last day of their roster (i.e. expect higher level of fatigue) 


If A-PiMod is to become a trusted (and relied upon) 3rd crew member, it needs to interact 
with the crew in a manner consistent with ‘normal’ human-human interaction [1]. Potentially, 
to be fully integrated in the ‘team’, it needs to engage in some form of ‘social’ interaction, and 
not only technical interactions [1, 2]. This latter issue has not been explored in A-PiMod and 
requires further consideration 


How emphatic a team member is automation? When and how can it articulate its concerns, 
and potentially, over-ride the crew member’s decisions? In most (but not all situations), the 
cockpit crew have the final authority and can veto automation. This approach reflects an 
‘adaptable systems’ logic. However, there may be situations (identified by the crew monitor- 
ing technology) where it is necessary for automation to ‘take charge’ to ensure flight safety (for 
example, situations of crew incapacity). Accordingly, three different levels of crew state moni- 
toring can be defined. For example, (1) passive support, (2) active support and (3) intervention/ 
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over-ride [1]. In this sense, A-PiMod represents an adaptive automation approach. Arguably, a 
fully adaptive automation approach requires modelling and assessment of both the crew and 
aircraft state [1, 2]. The A-PiMod system represents a significant advancement in relation to 
the modelling of crew state (i.e. touch, voice, gesture, use of multimodal displays and so forth). 
However, aircraft state modelling potentially necessitates integration with aircraft systems 
and wider AIM and ground systems. This has not been demonstrated in the A-PiMod project 


8. Innovation 


It is proposed that the Third Pilot/A-PiMod system (1) reflects a mix of the logic associated 
with adaptable systems and adaptive automation, while (2) also providing something new 
(i.e. multimodal cockpit concept). 


In relation to (1), we are 


e Providing a framework for crew-automation cooperation for all activities occurring in the 
cockpit involved in the completion of a flight 


e Extending concepts of assistance (as defined by adaptable systems), where the crew are in 
conceived to be in control all of the time 


e Utilising certain features of adaptive automation—that is, providing task support to the 
crew following an assessment of crew state (i.e. inferences about crew situation awareness 
and workload) 


In general, this new automation concept is predicated on concepts of partnership. The crew 
and automation are in charge together. As reported previously, in principle, the crew are in 
charge (concepts of professionalism and responsibility). However, there will always be par- 
ticular safety critical situations when automation can take charge (i.e. fully adaptive). 


In terms of (2), anew multimodal cockpit concept has been advanced. This enables improved 
assessment of the crew state/workload and provides a means to provide task and decision 
support information to the crew. Further, it allows for touch and voice interaction with cock- 
pit systems. Evidently, the application of multimodal in the cockpit is not new. However, the 
application of crew interaction data (i.e. crew voice and touch interaction in the cockpit), as 
an input to crew state monitoring is innovative. 


The third pilot has different modes of operation. This includes (1) passive monitoring, (2) 
active monitoring and (3) over-ride. It is anticipated that (1) and (2) will be the most typi- 
cal operational modes. That is, providing task and decision support to pilots based on an 
understanding/assessment of the crew state and specific workload requirements at a given 
point in time. In certain non-routine situations (3) will be required (i.e. fully adaptive). The 
partnership/third pilot concept is defined in relation to the combination of these three modes 
of operation. In the future, mode (3) might be become routine, whiles modes (1) and (2) might 
become less routine. Either way, the advancement of this concept will require considerable 
effort in relation to both development and certification [1]. 
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9. Assessment of benefits and safety quantification 


9.1. Assessment of benefits/impact 


The expected benefits/impact were validated through extensive field research including 27 
COP sessions, Validation Cycle 1 (VC1), Validation Cycle 2 (VC2) and evaluation with stake- 
holders at a demonstration day event [1, 2]. Overall, validation activities indicate that the 
‘Third Crew Member’ may yield many operational and safety benefits. As defined in Table 5 
below, these benefits can be grouped at different levels (i.e. cockpit, task, process and ATM System). 


It is anticipated that this third pilot approach will enhance flight safety, especially in abnormal 
situations. The third pilot will not remove human error. Rather, it will reduce it. This is largely 
attributable to improvements in error detection and error management. A cockpit that is designed 


No Benefit Benefit level 
1 Improved access to information—faster access Cockpit 
2 Improved mechanisms to interact with information (i.e. voice, touch and gesture) 


3 Provide variability in way of the automation control selection (i.e. via voice, touch and 
gesture) 


4 Better interaction between automation and Pilots— clear communication lines, clear 
understanding of who does what and, when/how transfer work between both 


6 Flexible and natural interaction with cockpit systems (MM) Task 
7 Decision support—how to proceed in safety critical situation and/or normal situation 


8 Provide task support to crew in relation to assessing the situation and deciding on a 
course of action 


9 Reduce workload in complex/high risk situations 

10 Pilots always know ‘what is going on’ (i.e. reliable situation awareness) 

11 Complementary ways in which to interact with information (i.e. MM interaction) 
12 Reduction in workload 


13 Improvement in information management— support information acquisition and 
information analysis 


14 Improvement in situation awareness 
15 Improvement in situation assessment 


16 Provision of new information (i.e. status of operation, status of automation and routing 
advice) 


17 Improvements in the elimination of and recovery from human error 
18 Support for error identification and broader TEM 

19 Error reduction 

20 Error mitigation 

21 Error avoidance—avoidance of safety critical situations 


22 Better interaction between automation and Pilots— clear communication lines, clear 
understanding of who does what and, when/how transfer work between both 


Adaptive Automation and the Third Pilot 19 
http://dx.doi.org/10.5772/intechopen.73689 


No Benefit Benefit level 


23 Completion of mission in accordance with the flight plan Process 
24 Safe landing 

25 Reduction in accident rate 

26 Help mitigate the consequences of survivable accidents 

27 Indirect support—flight punctuality 

28 Indirect support—management of delays 

29 Indirect support—more efficient flights (and allied impact on fuel consumption) 

30 Reduction in accident rate System/ATM 
31 Indirect support—flight punctuality 

32 Indirect support— management of delays 

33 Potential link to SESAR SWIM concept (i.e. sharing information across ATM network) 


34 Potential to extend to support single crew operations 


Table 5. Operational and Safety Benefits. 


with the A-PiMod approach in mind will extend automation capabilities in an adaptive way, to 
the extent necessary to support a safer flight. Potentially, such an adaptive automation approach 
might prevent many accidents. In this regard, we might consider the AF 447 accident [8]. As 
indicated in the accident analysis research with COP members and VC2 participants, all 19 par- 
ticipants indicated that A-PiMod would have played a significant role in preventing this accident. 


9.2. Safety quantification 


As reported previously, it is expected that the A-PiMod concept might lead to a reduction in 
the probability of fatal accidents by 43% [1, 21]. 


The assessment of safety impact concerns what has been advanced from a conceptual per- 
spective (i.e. A-PiMod concept), as opposed to the technological progress (set of tools devel- 
oped in the A-PiMod project, which reflects a particular implementation of the concept) [1, 
21]. It is these set of tools that were validated in simulator evaluations [1, 2, 4]. These tools can 
be viewed as a first technical instantiation of the A-PiMod system. The sophistication, scope 
and integration of the tools can be improved in future research and development [22]. Once 
implemented, the expected impact might be further assessed. 


10. Conclusions 


In the perfect world, A-PIMOD would provide an airline with the most experienced team (crew 
and automation), capable of dealing with any situation (routine, non-routine, safety critical), where 
skill level is constant, across all weather/routes/airports/time zones. The third Pilot helps avoid 
dramas. The Third Pilot/A-PiMod will not eliminate human error. Rather, it will reduce error 
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(human error is a reality and errors will always happen), and help manage them if they occur. 
In this way, the third pilot approach can be conceptualised in relation to ‘Smart Pilot Assistance’. 


A-PiMod will enable the automation system to account for pilots’ emotional and cognitive 
states. For example, normal, tired, stressed, overloaded and incapacitated. Together with a thor- 
oughly designed adaptive multimodal cockpit, this new technology will significantly improve 
the safety of flight, especially in abnormal situations and during situations of crisis management. 


It should be noted that the assessment of safety impact has been undertaken for the A-PiMod 
concept, rather than for its particular implementation as achieved in the A-PiMod project. 
The A-PiMod concept is an advanced adaptive automation concept for a multi-modal cockpit, 
wherein the automation is seen as a third pilot, and crew and automation continuously adapt 
to each other and the context, such that safety is maintained. In the course of the A-PiMod 
project a particular implementation of the concept was achieved by development of a set of 
tools, and these tools were used in validation experiments (i.e. validation cycle 1 and valida- 
tion cycle 2), in a flight simulator context. This set of tools can be viewed as a first technical 
instantiation of the A-PiMod system, and the sophistication, scope and integration of the tools 
can be improved in future research and development. 
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